Speculative fix for flaky order sync test #16330
Conversation
|
|
itsmeichigo
left a comment
itsmeichigo
left a comment
There was a problem hiding this comment.
I followed the instruction to run the test 1000 times. It failed in iteration 161 for me:
✘ Test syncOrder_when_already_syncing_doesnt_call_orderService() recorded an issue at PointOfSaleOrderControllerTests.swift:91:9: Expectation failed: (mockOrderService.syncOrderWasCalled → false) == true
Since I don't have enough context on this area, I hope @iamgabrielma can help take a look? 🙏
|
Observation is a really good framework for normal use, and kinda bad for testing. Unless anyone has any specific ideas to improve it, I'd vote for removing the test over spending more time on fixing it. |
There was a problem hiding this comment.
Changes LGTM. Interestingly running that specific test 1000 times failed once, while running the whole suite 1000 times (25k tests) never failed.
✔ Test run with 25000 tests passed after 35.358 seconds.
For a 0.1% chance I'd keep it, if this becomes a CI issue and starts to happen often I'd say let's just remove it.
Description
This test has been around for about a year, and there's no recent changes in the area, so I'm not really sure why it's started getting flaky... I also couldn't reproduce it reliably even running the tests thousands of times.
I couldn't see anything obvious, so this is a fairly speculative improvement, but removing the sleep should help.
Test Steps
In Xcode, right click the test suite diamond and select
Run Tests repeatedlyand run them 1000 times. I tried 5000 with no failures.Screenshots
RELEASE-NOTES.txtif necessary.